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Abstract 

This paper proposes a novel image fusion scheme based on contrast pyramid (CP) with teaching 
learning based optimization (TLBO) for visible and infrared images under different spectrum of 
complicated scene. Firstly, CP decomposition is employed into gvery level of each original image. 
Then, we introduce TLBO to optimizing fusion coefficients, which will be changed under teaching 
phase and learner phase of TLBO, so that the weighted coefficients can be automatically adjusted 
according to fitness function, namely the evaluation standards of image quality. At last, obtain 
fusion results by the inverse transformation of CP. Compared with existing methods, experimental 
results show that our method is effective and the fused images are more suitable for further human 
visual or machine perception. 

Keywords: image fusion, TLBO, contrast pyramid, optimization 

1. Introduction 

The spectrum is divided into IR, visible and UV spectroscopy, which have their own principles 
and characteristics of imaging, in different wavelength regions. In practical applications, 
multispectral resolution and high spatial resolution images are more widely used. Thus, the visible 
and infrared image fusion has become a hot topic in the field of image fusion. It is hoped that 
through some kind of image fusion techniques to fully complement both information, and access 
to more detail and accurate information on the scene, it is possible to locate the position of the 
heat source for identifying camouflage, testing resources, night vision and so on. 

Image fusion refers to a multichannel source image data acquired on the same object after the 
image processing and computer technology, enabling to maximally extract the favorable 
information of each channel, and finally integrated into a high-quality image, the purpose is to 
improve the utilization of image information, the accuracy and reliability of the computer 
interprets and the spatial resolution and spectral resolution of the original image, which will help 
to monitor. This paper relates to the visible and infrared images fusion with having a wide range of 
practical value in many areas, such as electronics testing, military combat, lunar exploration, 
medical image analysis and so on. 

Nowadays, there are already a lot of methods in the field of images fusion. For example, the 
intensity-hue-saturation (IHS) transposition, 1 principal component analysis (PCA), 2 Brovey 
algorithm, 3 wavelet-based methods, 4 ’ 5 ’ 6 ’ 7 or various pyramid algorithms such as contrast 
pyramid 8 ’ 9 ’ 10 ’ 11 , etc. They are relatively simple image fusion method and most widely used, but 
there are still various deficiencies. For instance, the former three fusion methods can lead to 
different degrees of spectral distortion and lower spatial resolution. The wavelet-based fusion 
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methods may generate “ringing” effects. CP-based fusion method can obtain better results, but its 
fusion coefficients are set by experience and are constants, there is very great blindness. We put 
forward a new method that TLBO 12 ’ 13 ’ 14 approaches to maximally optimize fusion coefficients on 
the basis of CP decomposition, changing the previous pattern of fixed coefficients selection, to 
achieve superior fusion effect. 

The rest of the paper is organized as follows. In Section 2, the decomposition and 
reconstruction of CP is briefly reviewed, and TLBO is simply introduced in Section 3. Then the 
application of TLBO for optimizing coefficients is described in detail in Section 4, Section 5 
explains the principle and fitness function of the proposed method. The effect images and analysis 
of experiments are showed in Section 6. Finally, the conclusion is summarized in Section 7. 

2. Decomposition and reconstruction of CP 

The pyramid of an image is calculated from bottom to top. It is decomposed each image into 
pyramid image sequence of multiple scales, high resolution image in the lower layer, low 
resolution image in the upper, the size of the upper image is a quarter of previous layer image’. In 
consideration of CP derived from Gaussian pyramid, every level of which is formed by the 
previous one through filtering, we first build the Gaussian window function. Suppose w denote 
5x5 window function, and satisfy the following constraints: 

(1) Separability 


w(m, n ) = w(m)w(n), m e [- m],ne\-2,2] 


(3) Symmetry 


4 
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'M=i 


vv(m) = w(— m) 


(3) 


ma 


(4) Parity items such as contribution 

w[—2)+w[2)+ w(o) = vt^-l)+w(l) (4) 

These constraints are used to ensure the performance of the low-pass, and the brightness of 
images is smoothness after shrinking and expanding, it does not appear the joint effect. We can 
instruct: w(0)=3/8, w(l)=w(-l)=l/4, 
mathematically expressed as: 


w(2)=w(-2)=l/16, the window function can be 
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(5) 


Gaussian pyramid as a set of image sequence, each level are made at the next higher level 
image through a low-pass filter and the down-sampling process. The frequency band of two 
adjacent is reduced by 1/8 rate, the image size by 1/4 rate decreases. Here we set the source image 
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isGo (/, ]), i < A/ 0 , j < N°, that is Gaussian pyramid’s bottom image, M°, N° is the number of 


columns and rows of the image respectively. G; represents the first /-level of Gaussian pyramid, 0 
< 1 < N(N is the total number of layers). To get G/, it must firstly take G;.i and the above 
constructed window function w convolve, and then the convolution result is done the 
down-samj 



where u, v, denote the number of columns and rows of the first / layer pyramid image, respectively; 



After winning Gaussian pyramid, it will be interpolated to enlarge. We introduce enlarge 
operator, that is interpolate new samples between the given numerical by using interpolation 
method. A certain level of the pyramid image is extended into the si former level image. 



Thus, it can solve the problem of these levels’ different in sample density. If carry out the 


following operation on G/, we can get the new image owning thdftoie|!ze as G/_|. 



A/,0 <i<u,0<j<v (7) 


where |f'+ m/2,j + n/2) is integer coordinates 
otherwise 


where G ( ■/ + " 

\ 2 ’ 2 , 


Every level of CP Ci, is a ratio of the corresponding Gaussian image at all levels. It is defined as 



( 8 ) 


Than we consider CP reconstruction, which is the inverse process of decomposition. According 
to the iteration of formulation (8), set / = N, N- 1, ..., 0, starting from the top of CP and turning 
down, we can get each layer of Gaussian pyramid: G„, G„_i, ..., Go. The reconstruction imageGo is 
the final fusion image. The formulation is as follows. 


o 



(9) 


where G/ is the same size as G; after G /+1 interpolated. 

The original images are decomposed as a series of decomposition layers, which have different 
solutions and spatial frequency, by the multiscale CP decomposition. At the same time, each of 
the decomposition levels reflects the contrast information of the corresponding spatial frequency. 
The fusion process is conducted in every spatial frequency layer respectively, so it may be aimed 
at the features and details of different frequency bands on different decomposition levels by using 
different fusion operators, for the purpose of highlight the features and details of the specific 
frequency band. It follows the principle that the human retinal image is processed in different 
frequency channel, so it may gain the fusion effect which is more close to the human visual 
characteristics. Through many literatures and simulation experiments, CP fusion method is better 
than general pyramidal decomposition and wavelet transform for visible and infrared images 
fusion. However, there are inherent weaknesses: the selection of weighted coefficients is mostly 
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based on experience, which is the coefficients values are prior set, it can be easy to cause unstable 
fusion effect. To improve this situation, we adopt the heuristic evolutionary algorithm, TLBO, 
which can search and estimate the fusion coefficient in certain decision space, adaptively modify 
weighted coefficients, in order to achieve the optimization of the fusion effect. 

3. Teaching learning based optimization 

TLBO introduced by Rao et al, 15,16,17 is novel mathematical model prepared and implemented 
for the optimization of a unconstrained non-linear continuous function, and is also a new variant 
of meta-heuristic optimization technique inspired by the natural phenomenon of teaching and 
learning process. Meanwhile, liking other nature-inspired algorithms, TLBO is also a 
population-based method that uses a population of solutions to proceed to the global solution. 
Assume one class C, which own the same merit level learners and is tough by the two different 
quality of teachers 7) and T 2 , obtains mean value M\ and M 2 , respectively. We define that the 
marks distribution curve of the class is a normal distribution, as formulation (10), in spite of 
having certain deviation with the actual, it still be help for analysis. We can draw as Fig. 1 that 
describes the above definition. 



( 10 ) 


Fig.l shows the two curves, one dotted line Curve-1 and the other solid line Curve-2, 
respectively represent the relation of between distribution density and marks under the two 
different teachers teaching 7) and To. The approximately middle vertical virgule corresponding 
scissa values are mean value M\ and M 2 . The terminal corresponding abscissa value of curves 
presents the best mark in the class, which is also the ability of teachers. The teacher own the 
most knowledgeable in the society, he or she will impart knowledge to every learner in this class, 
so that learners’ marks will be improved, the higher marks the learners obtain, the higher the mean 
values are. However, it must be noted that no matter how the teachers do their best effort, the 
mean is unlikely to be the level of the teacher in terms of knowledge. In order to improve the 
mean value, it requires a new teacher of superior quality. Furthermore, the learners can also leam 
from each other. So TLBO is composed by two basic operation, namely teaching phase and 
learner phase, teaching phase means learning from the teacher and learner phase means learning 
through the interaction between learners. Brief descriptions of the two phases are given as follows. 
3.1 Teaching phase 
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The mean value improvement of a class mainly relies on a good teacher, so selecting a 
knowledgeable teacher is a key of acquiring a good result. The teaching phase is also primarily 
responsible for the global search property of TLBO algorithm. In reality, most individuals hope 
the every learner’s mark always rises by acquiring supreme knowledge from the most 
knowledgeable person. Whereas it is not possible because of the random of the process, th 
teacher can only boost the average grade of a class to some extent. Suppose X oUti , X n ^, 
respectively denote the previous mark of learner X and his or her new mark through learning from 
a teacher at the /th iteration. How motivate X old} , to X newJ depends on the di~ 




existing and the new mean value, let M, be the existing mean value and M newi be 


value, Therefore the new solution is defined according to the following expression: 


X new . = X oldi + r(M newi -T f M .) 



where r, is a random number, and r, £ [0,1], T F is a teaching factor 
value to be changed. Because 7> is either 1 or 2, it can be designed as 


;ides how the mean 


T f = round [l + ran <*(0.1)1 


( 12 ) 


3.2 Learner phase 

The learner phase of TLBO algorithm is that learners enhance their knowledge by interaction, 
which also is help for improving the mean value. Every learner obtains new things from another 
learner randomly selected, when the selected one has more knowledge than him. So it is 
indispensable to judge the pros and cons of both before interaction. Where let X t be one learner, Xj 
be another which is different from^^former. By using the concept of learner phase, the 


improvement of the grade point of learners may be represented as follows: 
for i=l:N 
if f(Xd>f(Xj) 

Xnew, i ~ X oldt i + 



where /V is the number of iteration, r, is a random number, and r, e [0,1], fitness function f(X), and 

define the optimization problem as: maximize/(X), accept X new i if it gives a better value. 

4. The application of TLBO algorithm for optimizing coefficients 

In this section, the application of TLBO will be particularly described. The whole process is 
showed as follow the flowchart, Fig.2. 
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Fig.2 Flowchart for TLBO optimizing coefficients 
According to the flowchart, the procedure of TLBO optimizing coefficient is briefly explained 
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at the following steps. 

Step 1: Define the optimization coefficients and initialize the optimization group, the number of 
iteration. 

Firstly, giving a range (depending on the practical situation) of the coefficients, set up the value 
of samples in this range, randomly, and the samples are divided into the same size group. All of 
samples are called population. 


population = 


~x u 

x 2l . 

... X Gl 

X I2 

X 2 , 2 ■ 

... X C2 

x ls 

x 2S . 

... x cs 


_ximiz e/(X). 

len pick out the ( 
;acher. 






* /' < S, and i, j 
iubscript of I is ite. 


where define X t j as the value of the first j sample of the first i group, 1 5 
are integers. At the same time, let the number of the iteration as 7, a 
initialize ite = 1. 

Step 2: Deal with data of group, define fitness function: maxim 
Need to calculate mean value of every group M t . Then pick out the optimal average group X T 
according to the fitness function, which will act as the teachf 
Step 3: Teaching phase 

Let all samples of every group except the teacher leams from M T . Here there is a nested loop, 
express as follows: 
for i-l:G 
for j=l: S 
if ifT 

X new , i , j =X old , u + r l (X- l! £yfff 
endif 
endfor 
endfor 

then, judge whether the new solution is better than the existing or not, and decide to refuse or 
accept. 

Step 4: learner phase 

Unffor the first loop of the step-3, randomly choose Xh from groups, and i f h, begin to leam 
from interaction. 
if fm<f(M h ) 
for 7=1: S 

X new ,i,j mX M ,i,j + rfXij-Xbj) 

endfor 

else 

for 7=1: 5 


ether the 


A 


X new ,i, r X old , u + r, (X h ,j - X uj ) 

endfor 

endif 

Accept the new group if it gives better results, and execute step-3 and step-4 until the end of the 
first loop. It needs to pick out the best group again and go to step-2. 

Step 5: Finding out the final best result 
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Repeat step-2~step4 until ite — I. At the time of iteration termination, find out the local optimal 
sample from the best group. 

By the above steps, we can draw a conclusion that the application principle of TLBO to 
optimize coefficients is from global to local. Firstly, choose the optimal group, and then the 
optimal sample will be selected from the specific group. The selection strategy is very critical, not 
only decide the location of the optimal coefficients, but also accurate the final result. However, 
there are still some problem with worth of our attention, for instance: the number of samples and 
groups, it must be appropriate, if having too many samples of a group, it can lead to the slowly 
moving of global average, otherwise, it can occur that the selection of local is inadequacy. In 
addition, more groups can cause unnecessary waste of time and less can’t reach the deal solution. 

5. Image fusion algorithm based on CP with TLBO 

In this section, we explain how to fuse visible and infrared image based on contrast pyramid 
with TLBO (TLBO-CP), and set fitness function that evaluation the quality of fusion images. 

5.1 Principle of fusion based on CP with TLBO 

Let MG\ and MG2 be the visible image and the infrared images, respectively. They have the 
same size N x N as the original input images. And initialize the number of CP decomposition 
levels is /. Through the procedure of CP decomposition, the original images become the two 
images, CPM 1 and CPM2 simultaneously with the same size {Nil 1 ) x {Nil 1 ). The couple of images 
will wait for fusion until the optimal coefficients are chosen by TLBO according to the fitness 
function. Start iteration after initializing all basic variable, the detailed process of iteration has 
already been expressed at the Step-2~Step-4 of the section 4. Assume the optimal coefficients k\, 
& 2 - The CP decomposition images can carry out fusion. Formulation is as follows: 


CP _Fus = CPMlxk l + CPM2xk 2 


(14) 


5.2 Fitness function for assessing the quality of fusion images 

In order to better estimate image sharpness, we usually select more than one measure methods. 
Here are three measure techniques which are employed to as fitness function in this experiment, as 
follows: 

(1) Information Entropy (IE), is a concept of quantitative information, and the larger the value is, 
the more the image information is. Its corresponding mathematic expressions are also given as 
follows. 


a 



(15) 


where H denotes IE of the image, L is the total gray level of the image, p, denotes the ratio of the 
pixel number A, of the gray value i and the total pixel number N, namely 


p. =N./ N 


(16) 


(2) Average Grads (AG), can sensitively react the presentation skills of image to tiny detail, and 
appraise the image sharpness. Here g denotes AG and can be mathematically expressed as 



(17) 


where f(x,y) is the image function, M and N are the rows and columns of the image, respectively. 
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(3) Standard Deviation (SD), is an important index measure of image information richness, 
beyond that it can evaluate image contrast size. Its computation formulation is as follows. 




where S,d denotes SD, x is the mean value of the image pixel and x t is the pixel value of the image. 

6. Numerical experiments 

In the experiments, we firstly choose a couple of visible and infrared images of night scene as 
Fig.3 (a) and (b), they are used be 256 x 256 pixels and the gray-scale map. The other two 
groups of images have the same size as the first one and are colorized. Fig.4 (a) and (b) are the 
visible light image with band 1 (0.45 to 0.52 jum ) and the infrared thermal image respectively, of 
satellite images of Copenhagen City, Denmark. They were acquired on 26 May 1992. The last pair 
were respectively captured with a Sony Camcorder and a LWIR sensor, and showed that men and 
building with (Fig.5 (a)) and without (Fig.5 (b)) a smoke screen in cloudy weather. 

In this paper, we employ the PCA-based fusion method, wavelet-based fusion method and 
CP-based fusion method to compare with TLBO-CP method proposed. The number of 
decomposition levels of wavelet and CP transform is both three and the principle of their fusion 
also choose larger value. In the process of optimizing coefficients, the samples’ values of 
population are initialized between 0 and 1. 
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(e) jr 

Fig.3 (a)Original visual light image (b)Original infrared thermal image (c)PCA-based fusion result 


(d)Wavelet-based fusion result (e)CP-based fusion result (f)TLBO-CP fusion result 



(c) 


(d) 
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Fig.4 (a)Original visual light image (b)Original infrared thermal image (c)PCA-based fusion result 



(c) 


(d) 
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(e) jr 

Fig.5 (a)Original visual light image (b)Original infrared thermal image (c)PCA-based fusion result 
(d)Wavelet-based fusion result (e)CP-based fusion result (f)TLBO-CP fusion result (Color online 
only) 

6.1 Visual analysis 

The experimental results have been showed from Fig.3 to Fig.5. We can clearly find Fig.3 (c), 
Fig.4 (c) and Fig.5 (c) with seriously spectral information loss, which are based on PCA method 
fusion, especially Fig.5 (c), basically don’t see the target object. Then the Wavelet-based fusion 
images well display the visual effect of fusing visible and infrared images. But it is not hard to see 
details of the rough and faint hue from (d) of Fig.3~Fig.5. The last two images of every group 
respectively based CP fusion and TLBO-CP fusion have better fusion result. Comparing the fusion 
effect of the two methods, we can find that fusion results based TLBO-CP have higher intensity, 
better fuse the characteristics and details of the two original images, express more rich information 
level of fusion image, and highlight the objects, which will be more conducive to analysis in 
practical application. 

6.2 Quantitative analysis 

In order to evaluate objectively those methods mentioned above, we list the calculation results 
of fitness function described in section 5.2, as follows Tablel~Table3, corresponding to 
Fig.3~EIg;5. Although the numerical results of each experimental of TLBO-CP is unequal for 
randomness, it has been fluctuating in a small region. From the data of IE, AG, STD, IE value of 
the new method proposed is litter higher than the former even lower, but the values’ promotion of 
AG and STD are absolute superiority, which shows that images based TLBO-CP fusion have 
higher resolution and more abundant information, not only reflect the detail features, but also 
retain spectral information better. So our method also has better objective evaluation criteria. 
However, it is a problem to be introduced that run time is higher, because of the higher algorithm 
complexity of TLBO-CP. Suppose the number of iterations is /, the number of groups is G, and the 
number of samples of every group is S .So the number of the global coefficients selection 
becomes 2x/xG, and produce the same number of CP reconstruction and fitness function 
calculation, with producing 2 xIxGxS coefficients. Finally, there is still a local coefficients 
selection requiring the 5+1 times of CP reconstruction and fitness function calculation. 
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Table 1 

l Numerical results of Fig.3 



Images 

IE 

AG 

STD 

Run time 

Visual light color image 

3.4236 

2.856 

27.1248 

- 

Infrared thermal image 

4.6603 

4.8559 

32.8665 

A 

PCA-based fusion result 

4.8325 

5.9438 

39.9533 

0.2500 

Wavelet-based fusion result 

4.5077 

6.0738 

32.268 

0.0940 

CP-based fusion result 

4.3297 

6.7942 

37.1387 j 

0.0470 

TLBO-CP fusion result 

4.6996 

9.9983 

53.09M^' 

11.6719 


— 


Table 2 Numerical results of Fig.4 


Images 

IE 

AG 

STD 

Run time 

Visual light color image 

5.3444 

17.8197 

63.7882 

- 

Infrared thermal image 

4.4731 

6.5195 

68.5229 

_ 

PCA-based fusion result 

5.1868 

10.7996 

62.9817 

0.7960 

Wavelet-based fusion result 

5.2271 

18.51 

70.7156 

0.3440 

CP-based fusion result 

5tI|£> 

23.3201 

81.2657 

0.1570 

TLBO-CP fusion result 

4.4306 

27.8164 

95.9599 

11.4069 


le 3 Numerical results of Fig.5 


Images 
light color i 


r 0 


fared thei 


Visual light color image 
lermal image 
>ased fusion result 
'elet-based fusion result 
CP-based fusion result 
TLB O-CP fusion result 


IE 

AG 

STD 

Run time 

4.9224 

1.759 

52.9171 

- 

4.4127 

1.7589 

24.262 

_ 

4.9454 

1.7813 

53.6764 

0.0150 

4.4394 

2.3379 

30.185 

0.1090 

4.5823 

2.6309 

32.2092 

0.1090 

4.8984 

2.725 

48.7981 

11.3909 


7. Conclusion 

This paper presents a novel scheme of the visible and infrared images fusion, TLBO-CP. This is 
employing the nature-inspired optimization algorithm of TLBO to self-adaptively choose the 
fusion coefficients by fitness function evaluation. And through a lot of experiments verify the 
method excels the previous methods in both visual effect and objective evaluation criteria. In 
addition, it is feasible and simple to implement in comparison with existing efficient methods in 
practical applications. Because of the capability of real-time low, it is better to carry out the new 
fusion method on the system without requiring real-time. 
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Fig.2 Flowchart for TLBO optimizing coefficients 
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Table 1 

l Numerical results of Fig.3 



Images 

IE 

AG 

STD 

Run time 

Visual light color image 

3.4236 

2.856 

27.1248 

- 

Infrared thermal image 

4.6603 

4.8559 

32.8665 


PCA-based fusion result 

4.8325 

5.9438 

39.9533 

0.250fl^ 

Wavelet-based fusion result 

4.5077 

6.0738 

32.268 

0.0940 

CP-based fusion result 

4.3297 

6.7942 

37.1387 4 

0.0470 

TLBO-CP fusion result 

4.6996 

9.9983 

53.09^2 g 

11.6719 
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<e> 
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Table 2 Numerical results of Fig.4 



Images 

IE 

AG 

STD 

Run time 

Visual light color image 

5.3444 

17.8197 

63.7882 

- 

Infrared thermal image 

4.4731 

6.5195 

68.5229 

y 

PCA-based fusion result 

5.1868 

10.7996 

62.9817 

0.796fl^ 

Wavelet-based fusion result 

5.2271 

18.51 

70.7156 

0.3440 

CP-based fusion result 

5.1229 

23.3201 

81.2657 A 

0.1570 

TLBO-CP fusion result 

4.4306 

27.8164 

95.9599 

11.4069 
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<e> 
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Table 3 Numerical results of Fig.5 


Images IE AG STD Run time 

Visual light color image 4.9224 1.759 52.9171 
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Hig hl ights 

.This paper presents an optimized method for visible and infrared images fusion employing TLBO 
method. 

. Fusion coefficients are non-linear adjusted adaptively by fitness function. 

. Proposed method outperforms other methc 





